On-chip label-free cell classification based directly on off-axis holograms and spatial-frequency-invariant deep learning

We present a rapid label-free imaging flow cytometry and cell classification approach based directly on raw digital holograms. Off-axis holography enables real-time acquisition of cells during rapid flow. However, classification of the cells typically requires reconstruction of their quantitative phase profiles, which is time-consuming. Here, we present a new approach for label-free classification of individual cells based directly on the raw off-axis holographic images, each of which contains the complete complex wavefront (amplitude and quantitative phase profiles) of the cell. To obtain this, we built a convolutional neural network, which is invariant to the spatial frequencies and directions of the interference fringes of the off-axis holograms. We demonstrate the effectiveness of this approach using four types of cancer cells. This approach has the potential to significantly improve both speed and robustness of imaging flow cytometry, enabling real-time label-free classification of individual cells.

www.nature.com/scientificreports/ into account the amplitude profile of the cell as well. In off-axis holograms, the most dominant factor is the interference fringes. Priscoli et al. 27 have previously shown classification between two neuroblastoma cell lines based on the raw off-axis holograms. However, they did not take into consideration fringe frequency variations.
The angle between the sample and reference beams determines the frequency and orientation of the fringes. These are highly sensitive to small changes in the system environment and may vary between experiments and even during an experiment.
In this paper, we propose a spatial-frequency-invariant deep neural network that can perform cell classification directly on the off-axis holograms, which is an important step toward real-time label-free cell cytometry based on off-axis digital holographic microscopy. We demonstrate the new direct classification ability on off-axis holograms with variable spatial frequency, created in a semi-synthetic procedure, for four pairs of isogenic cancer cells, and then used the trained network, after transfer learning, to classify experimental holograms of two types of cancer cells during flow in a microfluidic channel chip.

Materials and methods
Cell types and sample preparation. We imaged and analyzed four isogenic cell lines, termed SW 28,29 , HS, TE, and WM. The SW cells are a pair of isogenic cancer cell lines: SW480 (ATCC CCL-228)-colorectal adenocarcinoma cells, and SW620 (ATCC CCL-227)-a metastatic form of these cancer cells collected from a lymph node of the same patient. The HS cells are a pair of a normal skin cell line, HS895.Sk (ATCC CRL-7636) and a melanoma cancer cell line, HS895.T (ATCC CRL-7637), taken from the same patient. The TE cells are a pair of a normal skin cell line, TE353.Sk (ATCC CRL-7761), and a melanoma cancer cell line, TE354.T (ATCC CRL-7762), are both taken from the same patient. The WM cells are a pair of a melanoma skin cell line, WM115 (ATCC CRL-1675), and a metastatic melanoma skin cell line, WM266.4 (ATCC CRL-1676), both taken from the same patient.
All the cells were grown in Dulbecco's modified Eagle medium (ATCC, SN. , supplemented with 10% fetal bovine serum (BI, SN. 04-007-1A). 2 mM L-glutamine (BI, SN. 03-020-1B) were added for the WM cell lines. The cells were incubated under standard humidity and temperature of 37 °C with 5% CO 2 until 80% confluence was reached. Before imaging, the cells were trypsinized for suspension and supplemented with a suitable medium.
Off-axis holographic imaging flow cytometry system. The cells' off-axis holograms were acquired using a custom-built Mach-Zehnder interferometer microscope, as depicted in Fig. 1. The Illumination laser was a low-coherence supercontinuum laser (Fianium, SC-400-4) coupled to an acousto-optical tunable filter (AOTF) set and emits a 650 ± 3.5 nm beam. The beam is split into two by a beam-splitter (BS1). One of the beams is directed with a mirror (M) to pass through the sample chip (S), modulating the imaged cell phase onto the light wavefront and through a microscope objective (MO, Newport 60×, 0.85 NA or Leica 440 40×, 0.66 NA). The other beam passes through a duplicate objective. The optical path of the beams is aligned to match by moving a retroreflector (RR1) accordingly. Both beams are combined with a second beam-splitter (BS2) through a tube-lens (TL, with a focal length of 150 mm or 200 mm with respect to the objectives) and interfere at the image plane at a small off-axis angle, where a CMOS camera (Thorlabs CC1545M), with 5.2 μm pixel size, records the image. www.nature.com/scientificreports/ Still images were captured by using an adhesive chamber (Scureseal, Grace Bio-labs, 800 μm depth) on a coverslip and refocusing the sample by moving the coverslip along the optical axis for each cell. While imaging a dynamic video, the cell can flow in and out of focus. To manage this problem, we used a shallow microchannel with a depth and width of 35 μm × 700 μm 30,31 , respectively. The shallow height reduces out-of-focus occurrences. The flow was generated by a pump (CETONI Syringe Pump neMESYS 290N) with low flow rates of 7-30 μl/hr. This rate allows the camera to utilize the entire field-of-view at a framerate of approximately 20 frames per second.
Dataset preparation. From the recorded holograms, we cropped out 290 × 290-pixel single-cell images, as dictated by the largest cell in the dataset. The experimental cell dataset size was 923 and 704 for the SW480 and SW620 cells, respectively; 127 and 138 for the HS895.Sk and HS895.T cells, respectively; 169 and 170 for the TE353.Sk and TE354.T cells, respectively; and 105 and 183 for the WM155 and WM266.4 cells, respectively. Each hologram went through an OPD reconstruction algorithm to reconstruct the OPD and the amplitude profiles recorded within the hologram, as follows: the 2-D Fourier transform of the hologram is calculated, one of the cross-correlation terms is cropped and undergoes an inverse 2-D Fourier transform. The resulting complex matrix is divided by a background complex wavefront matrix, produced by the same procedure, from a background hologram without any cells. Resulting in the complex wavefront (CWF) of the sample defined as: where is the central laser wavelength. For creating the training dataset, using the experimentally acquired CWF of each cell, we synthetically prepared new digital off-axis holograms with various off-axis spatial frequencies. This was done synthetically by adding the CWF of the cells with a reference wavefront and taking the absolute value square while setting the off-axis angle α and the fringes direction angle ϕ , defining the final off-axis synthetic hologram as: The off-axis angle α was chosen within a limit to resemble an authentic off-axis hologram and is related to the fringe frequency as f = sinα . The lower boundary of the frequency is set so that the cross-correlation terms of the hologram would not overlap the zero term in the Fourier plane, i.e., the lowest frequency of the point spread function (PSF) would not be less than 3/2 of the PSF size. The minimum spatial frequency is then: where M is the magnification, and NA is the numerical aperture of the objective lens. The upper-frequency boundary is dictated by the Nyquist frequency, calculated to be at least three pixels per cycle of the fringes. The new off-axis holograms were each normalized to be in the range of [0,1] and saved as a 24-bit PNG image file. The OPD profiles were normalized to the maximum value of the whole dataset and colorized using colormap "Jet". Examples from the dataset are shown in Fig. 2.
Deep neural network training and testing. After augmentations, the datasets contained 6,508 images for the SW pair, 2,120 images for the HS pair, 2,712 images for the TE pair, and 2,304 images for the WM pair. Each dataset was divided randomly into 80% for training, 10% for validation, and 10% for testing.
Several common CNNs were examined on our dataset (Google net, Resnet18, Resnet50, VGG16, VGG19, and MobileNetV2), where MobileNetV2 32 outperformed the rest. We used a pre-trained network based on the ImageNet dataset, included in MATLAB 2022b Deep Network Designer toolbox, and changed the input and final layers to fit two-class classification. We used Adam optimizer, with settings of 0.005 constant learning rate and batch size of 64.
On average, the networks started to converge within 3.5 epochs, and the maximum number of epochs was set to 20. The output network was chosen to give the best validation-to-loss value. All the training and testing were done on a parallel CPU (Intel i7 10700).
We trained 30 different MobileNetV2 networks, elaborated below as Experiments 1-30, with different settings, as elaborated in Table 1. To test the network ability to perform under different off-axis fringe spatial frequencies, we carried out eight tests only on the SW pair, with different off-axis angles ( α ) across its boundaries with jumps of 0.25° and constant fringe direction angle ϕ = 0 • . Next, we trained on images with constant off-axis angle α and random ϕ per each image. After proving the network ability to classify directly on off-axis holograms with a constant angle, we tested its ability to classify images with random α and constant ϕ = 0 • , and on both random α and ϕ . For the latter case, we increased the size of the dataset by using all eight augmentations on the SW cells.
Following those experiments, we used the pre-trained SW network as the basis for training on the other couples of cells with both random off-axis fringe spatial frequencies and angles. For comparison, we trained and tested the CNN on the OPD profiles of the cell pairs. Finally, we verified on the experimental SW-cell holograms the CNN ability for classification on real-world data.

Results and discussion
Tables 1, 2, 3 specify the performances of the network (accuracy, recall (sensitivity), specificity, precision, and area under the curve (AUC) of the receiver operating characteristic (ROC) graph) for the various experiments based on the testing dataset. All but the TE cells had results higher than 90% in all the metrics, with the TE metrics above 80%. Table 1 includes the experiments done on the synthetic holograms, Table 2 includes the experiments done on the OPD profiles, and Table 3 includes the experiments done on the original raw holograms. We did not find trends between Experiments 1-8 and between Experiments 9-16 to report, implying the network strength to cope with different fringes spatial frequencies. The main difference between the two sets of experiments is attributed to the randomness of the direction angle, which typically changes less than the off-axis angle.
It arises from Experiments 1-18 ( Table 1) that classifying through random angles (size or direction) is more challenging than performing classification based on one angle. In Experiment 18, when training on ImageNet pre-trained network with both random fringe spatial frequency and direction, the process failed, resulting in an accuracy of around 50%. To resolve this, we expanded the dataset to all eight augmentations and used Experiment 17 as the pre-trained network; thus, we achieved good classification results, albeit this test is the most complicated task due to its randomness of the parameters.
For comparison, we trained the CNN on the calculated OPD and compared the classification performance of the SW, HS, and WM cancer cell pairs (Experiments 22, 23, and 25, respectively) to these obtained in Roitshtain et al. 29 based on hand-crafted feature extraction followed by applying an SVM. The comparison of the performance metrics along with the ROC analysis are shown in Fig. 3, where the superiority of the CNN to the SVM is clear. In Experiments 18-21, the classification of the synthetic holograms with random angles, both α and ϕ , yielded similar or better results to the OPD-based experiments (Experiments [22][23][24][25]. For better comparison with Experiment 18, we ran Experiment 30 with all eight augmentations on the OPD profiles (Table 2). www.nature.com/scientificreports/ Next, we tested the networks trained on the synthetic holograms of the SW cells with the original recorded holograms and got poor results, most probably due to the high contrast of the fringes in the synthetic hologram versus the low fringes visibility in the real low-coherence holograms. Therefore, in Experiments 26-29 (Table 3), we used the captured raw holograms of the SW cells for training with different settings. Experiment 27 is trained, as in previous tests, on pre-trained ImageNet. Experiments 26 and 28 were trained using the network of Experiment 17 with different learning rates. In Experiment 29, we only used the small test set (10% of the dataset) to train the network, based on Experiment 17. The metrics for Experiment 29 were calculated on the whole training set (80% of the dataset), suggesting only a small change to the network weights was enough to utilize the semisynthetic trained network onto real holograms.
Finally, we used dynamic videos of flowing SW cells in a microfluidic chip and classified them in real-time. We classified them both directly on the raw hologram and on the calculated OPD (Experiment 30) for comparison. The classification was done using the networks trained on the in-focus still images in Experiments 26 and 27. The network in Experiment 26 was trained on the network of Experiment 17, and the network in Experiment 27 was trained on ImageNet weights. The confusion matrices and related metrics are summed in Table 4. Several classification frames are featured in Fig. 4b-d, an example video is presented in Supplementary Video 1. Classification directly on the holograms displayed higher performance than classification on the OPD profiles. The significant difference is most likely due to focusing problems during flow. While the OPD profile quality is very sensitive to focus due to fast changes in the phase around the image plane, the holograms, which are based both on the phase and amplitude, are less affected by changes in focus. The network in Experiment 26 yielded better results, as it was trained on synthetic holograms beforehand (Experiment 17). We chose Experiment 17 as the basis for Experiment 26 since throughout the entire videos (~ 14,700 frames), the fringe direction has not changed much, while the off-axis angle has fluctuated more than a full degree, as shown in Fig. 4e. Calculation of the OPD per frame takes approximatly 20 ms. Classification on the hologram takes 20 ms, while classification on the OPD is done in 25 ms. Thus, even when using deep learning hardware, we could increase the processing time by up to 2.3 times by classifying directly on the raw holograms rather than first processing the holograms to the OPD profiles and only then performing the classification.  Table 4. Results for real-time classification of flowing cells.

Conclusions
We have developed and trained a CNN to classify live cells during flow directly on the raw off-axis holograms with invariance to the off-axis angle size and direction, affecting the interference fringes of the hologram. First, we demonstrated our approach on semi-synthetic data, synthetic holograms generated based on experimental OPD profiles. We have shown the ability of a common CNN to classify a pair of isogenic cancer cells with different fringe frequencies and fringe directions, yielding similar classification performance to those obtained when classifying the OPD profiles. We have also compared the results of the CNN classification to those obtained by conventional machine-learning classification based on feature extraction. Testing was done by applying transfer learning on experimental holograms, improving their performances. Furthermore, we have tested the network on dynamic holographic videos of flowing cells. The hologram-based network succeeded in classification and was shown to be less sensitive to cell-focusing problems. Classifying directly on the holograms decreases the calculation time by 2-3 times, speeding up the throughput and robustness of the system. The proposed method provides an important step in the foundation of real-time label-free imaging flow cytometer, enabling actual sorting the further processing for the unlabled cells.